Rain detection apparatus and method

ABSTRACT

A rain detection apparatus includes a camera that views a surface and a processor that captures an image from the camera. The processor generates a signal indicative of rain on the surface from information contained in the captured image and optionally drives a surface cleaning apparatus in response thereto. The apparatus captures images focused at a plurality of distances. The processor includes an edge detector that detects edges visible in the captured image and a difference structure that calculates the difference between the number of edges visible between differing images. The edge detector disregards edges close to areas of light larger than the largest raindrop that is desired or expected to be detected. The apparatus optionally includes a backlight, and the difference in numbers of edges between frames with and without the backlight illuminated are used to distinguish between background features and rain on the surface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional patent application of U.S. patentapplication Ser. No. 11/035,677, filed Jan. 14, 2005, which was acontinuation of International Application No. PCT/GB03/03061, filed Jul.16, 2003, which claims priority from U.K. Patent Application No.0216483.8, filed Jul. 16, 2002. The disclosures of all of suchapplications are incorporated herein by reference.

BACKGROUND OF THE INVENTION

This invention relates to rain detection apparatus and method. Inparticular but not exclusively, it relates to a vehicle window cleaningapparatus. It also relates to a vehicle fitted with such an apparatus.

In one known situation, it has long been desired to automatically sensethe condition of a vehicle window in order to activate vehicle windowcleaning apparatus. An example would be to automatically activate thewipers and/or washers of a car's window (commonly referred to as awindscreen or windshield) should the window become rained upon or shouldit become otherwise obscured by dust, dirt or debris.

Prior art solutions to the problem have generally used infra red sensorsor sensors of the visible yellow type mounted on the appropriate window.However, these only sense the condition of a limited area of window andrequire close association with the window, which can cause problemsshould the window need replacing.

As well as cleaning of vehicle windows, it is also desirable to providefor the automatic cleaning of many other surfaces where obscuring mediasuch as rain drops, dirt or debris on the surface would cause problemsif it obscures the view through the surface. An example would be theglass or plastic lens of a headlamp or other light of a vehicle.

SUMMARY OF THE INVENTION

According to a first aspect of the invention there is provided a raindetection apparatus; comprising a camera adapted to view a region of asurface and a processor adapted to capture at least one image from thecamera, in which the processor is further adapted to generate at leastone signal to indicate the presence of rain on the surface frominformation contained in the at least one captured image.

By rain, we may include fog, mist, spray, drizzle and other forms ofmoisture or obscurating media that may form, fall or otherwise beincident upon the surface.

The apparatus may be adapted to capture images at a plurality ofdistances. In a particularly advantageous embodiment where the surfaceis a part of a vehicle, the apparatus may be adapted to sense thelocation of other vehicles or of lane markings on a road in which thevehicle to which the system is fitted is situated. In such anembodiment, the camera may be adapted to be focused at a plurality ofdistances. One of these distances may correspond to viewing the vehiclewindow and another to viewing the road ahead of the vehicle.Accordingly, the camera may be provided with a bi-focal lens, or amulti-focal lens. Alternatively, the camera may, in use, continuallychange focus, or a focusing mirror may be used or other means ofachieving a dual focus system may be provided. The camera may be adaptedto focus on the surface over a first region of its field of view, andthe road over a second region of its field of view.

Combining lane and vehicle detection apparatus with the apparatus hereindescribed is especially advantageous as both use the same hardware andeven elements of the same software and so use of such can be shared.

The surface may comprise a window, a window of a vehicle or perhaps asurface of a lamp such as a vehicle headlight. Such a system has theadvantages that it requires no contact with the surface, it can workwith visible light and can easily be adapted to sense any area of thewindow desired. In fact, it is preferable for the camera not to be incontact with the surface.

The surface may be the front window of the vehicle, commonly known asthe windscreen or windshield of the vehicle. Equally, it may be the rearwindow or any other window.

The apparatus may be adapted to control a vehicle window cleaningapparatus adapted to clean the surface, which may comprise one or morewipers, which wipe the surface of the vehicle window, preferably at anadjustable speed. The vehicle window cleaning apparatus may alsocomprise one or more washers, whereby washer fluid (perhaps water,possibly with added detergent or anti-freeze) can be sprayed onto thesurface of the vehicle window.

The surface may be a cover or lens of a vehicle light such as aheadlight. In this case, a camera may be provided inside the headlightbehind the cover or lens.

The system may be provided with fault warning means such as a lamp or asounder, which are adapted to alert a user of the system should a faultcondition occur.

The processor may include edge detection means adapted to detect edgesvisible in the at least one captured image. This may be performed bycalculating the spatial rate of change of pixel intensity in each of theat least one image. The spatial rate of change may only be calculated inone, preferentially horizontal, direction. Accordingly, vertical edgeswould be thereby detected. This may be performed by convolving theimage, typically with a Sobel-like mask or other suitable mask. Theposition of the edges may be determined as the positions at which thespatial rate of change exceeds a prescribed threshold. An example of asuitable edge detection technique is taught in WO 99/44173 in which theedges of lanes on a highway are detected.

The processor may also include counting means whereby the number ofdetected edges in the, or each, captured image may be counted. This maygive an indication of the amount of rain on the vehicle window. In analternative, the counting means may be adapted to count edges that arenew compared with a previous image. Mask memory may be provided, whichis adapted to record which edges are old, i.e. exist in the previousimage, allowing the processor to determine which edges are new.

The apparatus may be provided with a backlight, arranged to illuminatethe surface. The backlight may be adapted to cause any rain on thesurface to be substantially more visible to the camera for example bybeing mounted at an angle of less than 90.degree. to the surface. Theapparatus may be adapted such that rain is not visible to the camerawithout the backlight illuminated. This therefore allows successiveimages to be captured with and without the backlight illuminated. Thebacklight may comprise a plurality of individual illumination sources,or a single illumination source.

The processor may further comprise difference means adapted to calculatethe difference between the number of edges visible between imagescaptured with and without the backlight illuminated. This advantageouslyprovides for the apparatus to be able to discern between backgroundfeatures such as the headlights of oncoming vehicles visible with orwithout the backlight illuminated and rain and suchlike only visible (oronly detectable) with the backlight illuminated. Furthermore, onlycounting the number of edges in each image and then taking a differencerequires the use of less memory than a frame subtraction technique,where individual elements (such as pixels or individual edges) ofdiffering captured images are compared. It is also less prone to errorsdue the background of the images captured changing, as the position ofthe edges counted is irrelevant. The difference means may also subtractthe number of edges due to specular reflection of the backlight from thenumber of edges detected. This may be a predetermined constant number.

The backlight may be adapted to illuminate the surface with visible orinfrared light, and may illuminate the surface with a given range offrequencies. The processor may be adapted to control the activation ofthe backlight. The apparatus may further comprise a filter, typically anoptical bandpass filter, which preferentially allows the range offrequencies to pass.

In an alternative to the masks discussed above, the mask with which the,or each, captured is convolved may be adapted to disregard edges closeto areas of light larger than the largest raindrop that is desired orexpected to be detected. The mask may therefore ignore edges due to theheadlights of oncoming vehicles or specular reflection off the surface.The mask may comprise a central peak, and two troughs spaced on eitherside of the central peak in convolution space. The peaks and the troughsmay be of non-zero width, and may be separated from one another by areasof mask having a value of zero, or substantially zero. The peaks and thetroughs may be of opposite sign, and may each have linear rising andfalling edges and an optional constant maximum absolute value ofnon-zero width in convolution space.

Use of this mask may mean that the results of convolving the, or each,captured image with the mask for an edge corresponding to a backgroundfeature such as an oncoming headlight or the specular reflection of thebacklight off the surface are less significant than those for edges dueto rain. The apparatus may therefore include thresholding means wherebyany detected edges having a convolved value less than a predeterminedthreshold are not considered as edges.

The apparatus may be further adapted to detect the presence of mist onthe surface. By mist, we may mean mist, for or other small radius dropsof moisture. To this end, the processor may be adapted to calculate thedifference in intensity of the captured image in the area around, butnot including, the area of the image in which the specular reflection ofthe backlight is found, between images in which the backlight isilluminated and where it is not. The presence of a high difference inillumination has been determined to be indicative of the presence ofmist (large amounts of non-specular reflection). The difference may betaken of the average intensities around the specular reflection of thebacklight, or the difference itself may be averaged.

According to a second aspect of the invention there is provided avehicle fitted with the surface cleaning apparatus control system of thefirst aspect.

According to a third aspect of the invention, there is provided a methodof detecting the presence of obscurating material on a surface, themethod comprising capturing images of the surface and then calculatingone or more characteristics of the condition of the surface from thecaptured images.

The surface may comprise a vehicle window, or perhaps a cover for alight of a vehicle such as a headlamp. The images captured may bespecifically focused on the vehicle window.

In a particularly advantageous embodiment, the method also includesdetecting the positions of lanes or vehicles surrounding a vehicle frominformation contained in the at least one captured image. In such acase, the at least one captured image may be partially focused on thevehicle window and partially elsewhere, for example on the road ahead ofthe vehicle. Alternatively, successive captured images may be focused onthe vehicle window for use in determining the condition of the vehiclewindow and elsewhere, for use in lane and vehicle detection.

The images may be captured at a rate of substantially 100, 50, 20, 10,5, 2 or 1 per second, or every 1 s, 2 s, 5 s or 10 s. Equally, they maybe captured at any suitable rate. The images may be captured at a fixedrate substantially between 1 to 100 per second or one image between forexample 1 to 10 seconds. Optionally the capture rate may be varied inaccordance with a prescribed operational parameter of the system such asvehicle speed when the invention is associated with a motor vehicle.

The images may be of substantially all or part of the surface to besensed such as between 1% and 100% of the total surface. For example,they may be of at least 75%, 50%, 40%, 30%, 25% or 10%. The size may bedetermined so as to provide sufficient data for reliable processing.

The method may include the step of detecting edges visible in theimages. This may include the step of calculating the spatial rate ofchange of pixel intensity in each of the images. The spatial rate ofchange may only be calculated in one, preferentially horizontal,direction. Accordingly, vertical edges would be thereby detected. Thismay be performed by convolving the image, typically with a Sobel-likemask. The position of the edges may be taken as the positions at whichthe spatial rate of change exceeds a predetermined threshold.

Alternatively, the mask may be adapted to disregard edges close to areasof light larger than the largest raindrop that is desired or expected tobe detected. The mask may therefore ignore edges due to the headlightsof oncoming vehicles or specular reflection off the surface. The maskmay comprise a central peak, and two troughs spaced in convolution spacetherefrom. The peaks and the troughs may be of non-zero width, and maybe separated from one another by areas of mask having a value of zero,or substantially zero. The peaks and the troughs may be of oppositesign, and may each have linear rising and falling edges and an optionalconstant maximum absolute value of non-zero width in convolution space.

Use of this mask may mean that the results of convolving the, or each,captured image with the mask for an edge corresponding to a backgroundfeature such as an oncoming headlight or the specular reflection of abacklight off the surface are less significant than those for edges dueto rain. The method may therefore include the step of thresholding theconvolved values such that edges having a convolved value less than apredetermined threshold are not considered as edges.

The edges detected may be those of raindrops. By raindrops, we mayinclude fog, mist, spray, drizzle and other forms of moisture orobscurating media that may form, fall or otherwise be incident upon thesurface.

The number of detected edges in each image may then be counted to give,for example, an indication of the amount of rain on the vehicle window.This may be referred to as an “edge counting” method. In an alternative,so-called “edge boundary masking” method, edges that are new comparedwith a previous image may be counted. In the case where there is noprevious image, all detected edges may be counted as new. Preferably arunning total of new edges is kept. A mask may be used to record whichedges are new. With each image, the mask may be updated. Updating themask may involve marking the areas around new edges as old. The areamarked as old with each new image may be the new edges dilated by apredetermined amount.

These two alternatives provide two different ways to interpret the edgesvisible on the vehicle window. The first alternative is simple andrequires little processing power whilst the second alternative is moreable to cope with shifting raindrops and movement of background images.Furthermore, in the second alternative the number of new edges in eachimage may be used to determine the rate at which it is raining and therunning total may be used to determine the amount of rain on the vehiclewindow.

The method may further comprise the step of taking the differencebetween the number of edges visible between images captured with andwithout a backlight illuminated. This advantageously provides for themethod to be able to discern between background features such as theheadlights of oncoming vehicles visible with or without the backlightilluminated and rain and suchlike only visible (or only detectable) withthe backlight illuminated. Furthermore, only counting the number ofedges in each image and then taking a difference requires the use ofless memory than a frame subtraction technique, where individualelements (such as pixel s or individual edges) of differing capturedimages are compared. It is also less prone to errors due the backgroundof the images captured changing, as the position of the edges counted isirrelevant. The difference means may also subtract the number of edgesdue to specular reflection of the backlight on the surface from thenumber of edges detected. The number of edges due to specular reflectionof the backlight on the surface may be a predetermined constant number.

The method may further include the step of controlling vehicle windowcleaning apparatus according to the sensed vehicle window condition.Advantageously, the vehicle window cleaning apparatus may include one ormore wipers, which wipe the vehicle window, or washers, which spray thevehicle window with fluid.

In a further development of the system, the average size and or densityof the detected media may be further used to control the cleaningapparatus. In the case that the vehicle window cleaning apparatus is oneor more wipers, such control may include calculating average raindropsize and density. The average size and density of raindrop may then beused to generate a signal indicative of the speed at which the wiper orwipers are to run. An intermediate factor may be calculated for each ofthe density and average raindrop size. These two factors may then bemultiplied together to form a wiper control signal. Each factor may bebetween 0 and 1. Each factor may be zero below a first value of eitheraverage raindrop size or density respectively and 1 above a second valueof average raindrop size or density respectively. Between first andsecond values, the factors may linearly increase with average raindropsize or density. The factors may increase from an initial factor valueto a terminal factor value. Preferably, the initial factor value isbetween 0 and 1 and the terminal factor value is 1.

Accordingly, the wiper or wipers will only be operated should bothaverage size and density of raindrops exceed certain limits, and will belimited to running at a maximum speed indicated by a wipe control signalof “1”.

The limits at which wipers (or other cleaning apparatus) are operatedmay be adjustable, and may be controlled by a user such as a driver of avehicle. This can also be applied with the basic edge detection method,with the number of edges that need to be obscured before cleaning beingadjustable or otherwise user defined.

The vehicle window cleaning apparatus may be activated with no directuser interaction. This enables the user of a vehicle to concentrate ontheir driving or other such tasks without the need to directly operatethe apparatus.

In the case where the amount of rain is determined by counting newedges, at least one of any running total and the mask may be reset whenthe vehicle window is cleared. This may be sensed by determining whenthe wiper or wipers have wiped the vehicle window. Alternatively, thetotal number of edges visible ignoring the mask may be counted andshould a sudden decrease occur the vehicle window is considered cleared.

This step is included as the edge boundary masking method assumes thatedges never disappear and hence the total number of edges alwaysincreases.

The amount of rain may be integrated over time, such that the amount oftime that rain has been-present on the surface is taken into account.The method may include the step of causing the surface to be cleanedwhen the integrated value exceeds a predetermined limit. Thisadvantageously allows small amounts of rain to be wiped after a maximumamount of time.

Furthermore, the method may include determining whether at least one ofdirt, dust, ice, frost and mist exist on the vehicle window. If one ormore does, then appropriate steps such as activating the vehicle windowcleaning apparatus, activating a vehicle window heater, activating avehicle heater or activating a vehicle air conditioning system may betaken. The method may include reading the temperature external to thevehicle, typically with an onboard external temperature sensor of thevehicle. The method may therefore include activating the vehicle heateror demister in response to the measured temperature, preferably incombination with the detected window condition.

By mist, we may include haze, dew, fog or other small radius drops ofmoisture. To this end, the method may include the steps of calculatingthe difference in intensity of the captured image between images inwhich a backlight is illuminated and where it is not in the area around,but not including, the area of the image in which the specularreflection of the backlight is found. The presence of a high differencein illumination has been determined to be indicative of the presence ofmist (large amounts of non-specular reflection). The difference may betaken of the average intensities around the specular reflection of thebacklight, or the difference itself may be averaged.

Also, the method may include the step of detecting from the smearpattern visible on the vehicle window after a wipe has passed thecondition of the wiper. This may involve a Hough transform of the smearpattern. Such smear patterns may also be used to indicate dirty water onthe vehicle window and to take appropriate action. The system may beable to differentiate between rain, snow and ice.

Other advantages of this invention will become apparent to those skilledin the art from the following detailed description of the preferredembodiment, when read in light of the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows schematically a sensing system according to the presentinvention;

FIG. 2 shows an image captured by a video camera;

FIG. 3 is a flow chart depicting a first method of estimating the amountof rain on a windscreen;

FIG. 4 is a flow chart depicting a second method of estimating theamount of rain on a windscreen.

FIG. 5 is a graph showing the relationship between average raindrop sizeor density and the intermediate, scaled, value;

FIG. 6 shows the steps taken in calculating the wiper demand signal fromthe edge count out part;

FIG. 7 shows a mask for use in convolving the captured images;

FIGS. 8 a, 8 b and 8 c show the effect of the presence of mist on thewindscreen;

FIG. 9 shows a Frame Subtraction technique according to the prior art;

FIG. 10 shows an Edge Subtraction technique according to the presentinvention;

FIG. 11 shows the steps carried out in this method in common with a lanedetection technique;

FIG. 12 shows the steps carried out in order to detect the presence ofmist on the windscreen; and

FIG. 13 shows a wiper control strategy.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 shows a sensing system according to the present invention. Thesystem comprises a camera 10, which views the scene visible through awindscreen 16 of a car. The camera is connected to a processor 12, whichis adapted to capture images from the camera 10. Also provided arewipers 18 (only one of which is shown in FIG. 1) which are, as is commonin the art, in the forms of arms, which can be driven in an arcuate pathover the outer surface of windscreen 16 by wiper motor 14. Wiper motor14 is responsive to signals generated by processor 12. Further providedis washer pump 20, which is optionally adapted to respond to signalsgenerated by processor 12 to pump washer fluid (usually water, perhapswith detergent or anti-freeze additives) onto windscreen 16 throughnozzles 22 at an appropriate safe time. The system is also provided witha fault warning light 24, which indicates the presence of a faultcondition.

The signals which are generated by the processor 12 in order to controlthe actuation of wiper motor 14 and optionally the washer pump 20 arecalculated at least partly in response to the images captured from thecamera 10. These shall now be described hereinbelow for a more specificembodiment of the invention, which uses an edge detection algorithm todetermine the amount of rain on the windscreen.

This embodiment also uses the apparatus described above. Camera 10 ismounted inside the body of a car on a mounting bracket (not shown)attached to the roof of the car. The camera views the scene in front ofthe car through the windscreen. The system, in addition to sensing rainon the windscreen, is adapted to sense the position of other vehiclesand lane markings etc. In order to facilitate this, a bifocal lens 24 isfitted to camera 10. The images captured from the camera 10 aretherefore split horizontally as exemplified in FIG. 2.

An optional light source, such as IR/LED 21 aimed at back-lighting thewindscreen may be provided. This may be enabled when detected orotherwise determined ambient light levels fall below a prescribed level.The effect or the use of the LED would be to highlight the edges of theobscurating media present on the image scene so as to enable the controlsystem to perform more reliably. An optical bandpass filter 11 is alsoprovided, which increases the dominance of the backlight wavelengthagainst the background scene.

FIG. 2 shows an example image 30 captured by the camera 10. The image 30is split into a top part 32 and a bottom part 34, divided by a dividingline 42. This line 42 is not normally visible in the image but has beendepicted for purposes of clarity. The top part 32 has been focused onthe outer surface of windscreen 16 on which raindrops 36 are visible. Inthe bottom part 34 of the image, the image has been focused on the roadahead whereby features such as road edges 38, lane markings 40 and othervehicles 44 can be seen. The processor can then use any of the methodsknown in the prior art to calculate the position of these features.Unless specifically mentioned, when referring to the images we shallhenceforth refer to the top part 32 of the captured images as the partrelevant to the amount of rain on the windscreen.

In order to calculate the need to wipe the windscreen 16 the steps of afirst method, shown in FIG. 3, are taken. Firstly, at step 100, theprocessor 12 periodically captures the top part 32 of images viewedthrough the camera 12. This occurs N times a second, although it isenvisaged that it may be more or less frequent. By way of example only,N may be in the range 15 to 40 times a second.

The next step 102 is for the captured images to be convolved with amask, such as a Sobel-like mask. The resultant values represent thespatial rate of change, in the horizontal direction, of the pixelintensity of the image. Taking a threshold 104 of these values (that is,determining the points in the image at which the convolved values exceeda pre-determined value) then gives an indication of where the verticaledges of raindrops 36 lie.

This method of edge detection is used as it is commonly in use in laneand vehicle detection systems, such as is known from WO 99/44173.

An alternative edge detection scheme replaces convolution with theSobel-like mask with convolution with the mask 51 depicted in FIG. 7 ofthe accompanying drawings. This consists of a central peak with twoouter troughs to either side (in convolution space). The peaks andtroughs have linear leading and rising edges and may have areas ofconstant value between them or at their absolute maxima. Convolutionwith this mask gives high values for edges, but edges that occur closeto large areas of light are penalized. Accordingly, vehicle headlightsand the specular reflection of the backlight 21 off the vehiclewindscreen 16 are disregarded as not being raindrops. The width andposition of the peaks (as in trace 51 a of FIG. 7 of the accompanyingdrawings) can be adjusted in order to tune the raindrop detection andheadlight elimination characteristics.

However, the person skilled in the art will recognize that any suitableedge detection algorithm could be used.

In the next step 106, the number of edges is counted. This gives anindication of the amount of rain present on the windscreen 16particularly where day and/or night screens are being viewed and wherethe glare from oncoming headlights would need to be accounted for. Useof multiple mask types, enables independence upon the prevailing ambientscene conditions to be complemented using a simple suitability logic.

In an improvement to this step shown in FIG. 10 of the accompanyingdrawings, the backlight 21 may be employed to increase the visibility ofthe edges of raindrops. The backlight 21, camera 10 and edge detectionmethod can be arranged such that edges of raindrops are only detectedwhen the backlight 21 is illuminated. Images are captured without 203and with 207 the backlight 21 illuminated, and the number of edges ineach of the images calculated. Images captured without the backlightilluminated will only show background details such as passing sceneryand headlights of oncoming vehicles. Illuminated images will show thesame features, plus highlighted raindrops and the specular reflection ofthe backlight in the windscreen. The number of edges for non-illuminatedimages then is subtracted 209 from the number of edges for theilluminated images. This value can be used to control 211 the wipers ofthe vehicle in the manner described below.

Accordingly, such artifacts as headlights and passing scenery can beaccounted for without the complications and memory requirements ofcomparing frames, as in the prior art method shown in FIG. 9 of theaccompanying drawings. The prior art method requires frames to becaptured without 202 and with 206 the backlight illuminated and thepixel values for each frame are subtracted from one another. Edges aredetected 210 in the resultant image and wiper control 212 carried outfrom the determined number of edges. This requires much more memory andprocessor time than the method of FIG. 10 of the accompanying drawings,as entire frames are compared rather than single numbers of edges.

If the specular reflection of the backlight has not been removed fromthe detected edges by the alternative mask 51, then these reflectionscan be accounted for by subtracting a further, predetermined amount fromthe number of edges for the illuminated images.

In an alternative embodiment, the first method of determining the amountof rain on the windscreen is replaced by a second method depicted inFIG. 4. Steps equivalent to those in the first method have beenindicated by the same numeral increased by 50.

In this method, the images are periodically captured 150, convolved andthresholded 154 as in the first method. The resultant edges are thencompared 160 to a mask 170 stored in a memory 13 associated with theprocessor 12. The mask 170 indicates which part of the scene is alreadyconsidered to be an edge of a raindrop, and so initially the mask 170will be blank. The comparison hence shows the raindrops which haveformed edges since the mask 170 was last updated. The number of thesenew edges is counted 162 and added 164 to a running total 172. Thistotal 172 will initially be zero.

The mask is then updated 166 by dilating the new edges by a dilationamount of, say, 5 pixels and marking the dilated edges on the mask 170as having been seen. This is a reasonable compromise between mistakingold edges that have simply moved slightly with respect to the camera 10as new edges and mistakenly ignoring new edges that form. Of course,this dilation amount can be varied to achieve the best results.

The method then repeats with the next image captured 150 from the camerausing the updated mask 170 and total 172. It can be seen that the totalwill never decrease merely from adding the number of new edges.Accordingly, it is necessary to reset the total 172 and mask 170 when asignal from the wipers 18 indicates the screen is freshly wiped.Alternatively, the first method may be employed to give a reading oftotal number of edges visible and the total 172 and mask 170 could bereset if a large, sudden, decrease in the total number of edgesaccording to the first method was seen.

To estimate the amount of rain on the windscreen, it is possible to usethe total number of edges from either method. Additionally the number ofnew edges in each image as calculated by the second method can be usedas an indication of the rate of fall of rain. When a prescribed oradapted threshold number of edges is reached, an enabling control signal174 may be generated causing the wash/wipe cleaning apparatus to beactivated to sweep and clean the windscreen.

In a preferred alternative depicted in FIG. 13 of the accompanyingdrawings, the number of edges is integrated over times such that smallamounts of rain are cleaned off the windscreen after a maximum period.The number of edges for a given frame is counted 250 according to any ofthe methods described above. If this value is greater than apredetermined “noise” threshold 252, then the number of edges is added254 to a running total. If not, the running total is reset to zero 256.If the running total exceeds a predetermined “wipe” threshold 258, thewipers are activated 260 and the running total reset 262. Otherwise, therunning total is kept 264 for the next captured image. The method thenrepeats.

In a further optional refinement, the processor 12 may then use theseaforementioned outputs to calculate average size 210 and density 212 ofraindrops on the windscreen 16, as shown in FIG. 6. These are then eachscaled 214,216 to give a value between 0 and 1 according to FIG. 5.Below a certain range 200 of raindrop size or density 203 the scaledvalue 202 is constantly zero; above the range 200 the scaled value 202is constantly 1; and within the range 200 the scaled value 202 increaseslinearly with raindrop size or density 203 from a minimum scaled value201 (which is between 0 and 1) to 1. These two scaled values aremultiplied together 218 to give a wiper demand signal 220 between 0and 1. This signal 220 indicates the fraction of maximum wiper speed atwhich the wipers 18 should run, where 0 is no wiper action and 1indicates maximum speed, providing for the wipers 18 only operatingabove a certain threshold of both size 210 and density 212 of raindrops.

The method used in this system is advantageously combined with a vehiclelane and object detection system as the steps of capturing 100, 150 andconvolving 102, 152 images (grouped as 108 and 158 in FIGS. 3 and 4) arealready performed by such a detection system and hence the addition of arain drop detection system as described herein does not unduly increasecomputational requirements. As to the choice between first and secondmethods, the first method requires less computational power and memory.However, it will produce much noisier results as raindrops shift withvehicle vibrations and so on, and as the unfocused scene behind thewindscreen changes. As the second method uses an Edge Boundary Maskingalgorithm, it is less affected by these problems but requires more inthe way of memory 13 and processor 12 use.

It is also appreciated that whilst with the cameras which are envisagedto be used in this system will not clearly capture the image of apassing wiper 18 and so the system will not recognized as an edge.However, cameras that are capable of sharply capturing moving wipers maybe used in which case correction must be made such that the wiper edgesare not unduly counted.

In another improvement, the system is adapted to detect the presence ofmist on the windscreen. This can be demonstrated with reference to FIGS.8 a to 8 c of the accompanying drawings, which depict views capturedfrom the camera 10, and FIG. 12 of the accompanying drawings, which showthe steps taken. In normal view, without the backlight illuminated 220(FIG. 8 a of the accompany drawings), the camera views the backgroundscene and any raindrops on the windscreen. The average intensitysurrounding the specular reflections of the backlight is measured 222.The backlight 21 is then turned on 224. In the lack of mist,illuminating the backlight 21 (FIG. 8 b of the accompanying drawings)does not have any great effect on the average intensity surrounding thespecular reflections 53 of the backlight. However, if mist is present(FIG. 8 c of the accompanying drawings) then large amounts of diffuse,non-specular, reflection occur, and the intensity in the regionsurrounding the specular reflections of the backlight is greatlyincreased. Therefore, if the average intensity around the specularreflections 53 is calculated 226 and the difference between the twomeasured average intensities taken 228, the presence of mist can bedetected as a large difference. This can then be monitored by thesystem, which can activate 230 demisting heaters, blowers and so on asappropriate.

FIG. 11 of the accompanying drawings shows how the system hereindescribed is advantageously used in combination with a lane detectiontechnique. The video camera 10, the frame grabber 19 (that part of theprocessor 12 adapted to capture images from the camera 10) and thevertical edge detection 17 are all identical for both the rain detectiontechnique described herein and a lane detection technique. The sameprocessor may therefore run the same instructions on the same data. Onlythe thresholding 15, edge counting 13 and wiper control 11 add to theprocessing and hardware requirements.

In accordance with the provisions of the patent statutes, the principleand mode of operation of this invention have been explained andillustrated in its preferred embodiment. However, it must be understoodthat this invention may be practiced otherwise than as specificallyexplained and illustrated without departing from its spirit or scope.

1. A mist detection apparatus comprising: a camera adapted to view asurface; and a processor adapted to capture at least one image from thecamera and is adapted to generate at least one signal indicative of thepresence of mist on the surface from information contained in the atleast one captured image, the apparatus further comprising a backlightadapted to illuminate the surface, wherein the processor is arranged soas to calculate a difference in intensity of the captured image in afirst area around a second area of the image in which a specularreflection of the backlight on the surface is found, between capturedimages in which the backlight is illuminated and where it is not, theprocessor further being arranged to generate the signal based upon thedifference in intensity.
 2. The mist detection apparatus of claim 1wherein the first area does not include the second area.
 3. The mistdetection apparatus of claim 1 arranged so as to activate at least onefrom the group comprising a vehicle window heater, a vehicle heater, anda vehicle air conditioning system in response to the detection of miston the surface.
 4. The mist detection apparatus of claim 3 arranged soas to activate a vehicle blower in response to the detection of mist ofthe surface.
 5. The mist detection apparatus of claim 1 wherein theprocessor is arranged to determine the average intensities in the firstarea.
 6. The mist detection apparatus of claim 1 wherein the processoris arranged to determine an average of the difference in intensity overthe first area.
 7. The apparatus of claim 1 wherein the backlight isarranged to illuminate the surface with a range of frequencies of light,and wherein the apparatus comprises a filter which preferentially allowsthe range of frequencies to pass.
 8. A method of detecting the presenceof mist on a surface comprising the steps of: capturing images of thesurface with a backlight illuminated and not illuminated; and thencalculating the presence of mist on the surface from the capturedimages, the method including the further steps of calculating adifference in intensity of the captured image between images in whichthe backlight is illuminated and where the backlight is not illuminatedin a first area around a second area of the image in which a specularreflection of the backlight is found.
 9. The method of claim 8 whereinthe first area does not include the first area.
 10. The method of claim8 further comprising the steps of activating at least one of the groupcomprising a vehicle window heater, a vehicle heater, and a vehicle airconditioning system in response to the detection of mist on the surface.11. The method of claim 10 further comprising the step of activating avehicle blower in response to the detection of mist of the surface. 12.The method of claim 10 further comprising the step of determining theaverage intensities in the first area.
 13. The method of claim 10further comprising the step of determining an average of the differencein intensity over the first area.